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PROCEDURES  FOR  VALIDATING  SKILL  QUALIFICATION  TESTS 
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!•  improved  procedures  for  validating  Skill  Qualification  Tests  have 
been  prepareda'for  your  use'T*^The  improved  procedures  are  based  on 
experience  gained  during  development  of  SQT  by  the  schools  in  EPMS 
Implementation  Group  I and  review  by  ITEG.  An  outline  of  the  new 
validation  procedures  is  presented  at  Inclosure  1.  The  details  of  the 
new  validation  procedures  are  presented  in  this  letter  which  incorporate 
the  most  current  guidance  and  are  at  Inclosure  2.^The  November  1975 
version  of  the  validation  procedures  is  obsolete. 


2.  Four  major  lessons  were  learned  during  the  development,  validation, 
and  ITEG  review  of  the  first  set  of  SQT.  These  are: 


a.  ITEG  review  of  the  Scoreable  Units  occurred  too  late  to  provide 
timely  guidance  to  the  Test  Development  Activities  (TDA)  on  the  content 
of  the  Scoreable  Units.  Accordingly,  new  procedures  are  set  forth  in 
this  letter  (paragraph  3,  below)  to  have  ITEG  review  the  SQT  Scoreable 
Units  and  components  prior  to  tryout  on  soldiers. 


b.  The  validation  procedures  presented  in  the  November  1975  version 
were  not  suitable  for  low  density  MOS.  New  procedures  have  been  devel- 
oped that  can  be  used  for  all  MOS,  both  high  and  low  density.  These 
procedures  are  described  in  paragraph  4 below. 


c.  Representative  soldiers  for  use  in  trying  out  the  SQT  in  the 
field  were  not  available  for  many  MOS.  The  sequence  of  validating  and 
reviewing  Scoreable  Units  has  been  changed  to  provide  more  useful  infor- 
mation to  both  the  TDA  and  ITEG.  These  changes  are  described  in  para- 
graph 5. 

d.  The  purpose  of  each  phase  of  the  validation  procedure  was  not 
fully  understood,  and  as  a result  the  validation  procedures  sometimes 
were  not  modified  to  adapt  to  local  conditions.  The  reasons  for  comple- 
ting each  step  of  the  validation  are  contained  in  the  succeeding  para- 
graphs of  this  letter. 


3.  ITEG  will  review  Scoreable  Units  before  they  are  tried  out  on  sol- 
diers. The  sequence  of  reviewing  and  validating  hands-on  and  written 
Scoreable  Units  is  presented  at  Inclosure  1.  The  significant  reviews  by 
ITEG  are  as  follows: 

a.  The  first  review  is  of  the  SQT  Plan  which  is  submitted  to  ITEG 
for  approval  prior  to  the  construction  of  Scoreable  Units.  The  SQT  Plan 
lists  the  tasks  covered  by  the  SQT  and  the  Component  (written,  hands-on, 
or  performance  certification)  to  which  a task  is  assigned  for  testing. 
Included  in  the  submission  of  the  SQT  Plan  is  a draft  of  the  Soldier's 
Manual.  (Detailed  procedures  for  preparing  and  submitting  the  SQT  Plan 
are  contained  in  a separate  document.)  ITEG  will  complete  their  review 
of  the  SQT  Plan  within  10  working  days  after  receiving  it.  In  the 
meantime,  TDA  will  begin  construction  of  the  SQT  sample  (see  para  3b). 

b.  After  the  SQT  Plan  has  been  approved  by  ITEG,  a sample  of  Score- 
able  Units  is  sent  to  ITEG  for  review.  The  sample  is  to  consist  of  5 or 
6 written  Scoreable  Units,  2 hands-on  Scoreable  Units  (if  the  SQT  con- 
tains a hands-on  component)  and  all  performance  certification  Scoreable 
Units  (if  the  SQT  contains  a performance  certification  component). 

Along  with  the  sample  of  Scoreable  Units,  the  SQT  Notice  and  the  task 
analyses  corresponding  to  those  Scoreable  Units  must  also  be  submitted. 
The  task  analyses  will  have  the  essential  and  key  steps  identified. 
(Essential  steps  are  those  behaviors  necessary  to  perform  the  task,  and 
key  steps  are  the  behaviors  that  have  serious  consequences  to  personnel 
or  equipment,  or  are  frequent  sources  of  failure  in  performing  the 
task.)  ITEG  will  review  the  content  of  Scoreable  Units  to  judge  whether 
the  content  of  the  Scoreable  Units  adequately  covers  task  performance. 
ITEG  will  reply  to  the  TDA  within  10  working  days  after  receiving  the 
sample  Scoreable  Units.  If  the  sample  is  not  approved  by  ITEG,  it  will 
be  returned  to  the  TDA  for  revision  and  a second  sample  would  then  be 
prepared  and  submitted  for  ITEG  review.  In  the  event  that  the  second 
sample  is  not  satisfactory,  direct  contact  between  ITEG  and  TDA  repre- 
sentatives will  be  required.  This  paragraph  and  3a  above  corresponds  to 
Phase  0 in  Inclosure  1. 

c.  After  receiving  approval  from  ITEG  that  the  content  of  the 
sample  is  satisfactory,  the  entire  written  and  hands-on  components  will 
be  developed.  Each  of  the  hands-on  and  written  Scoreable  Units  will  be 
tried  out  locally  with  five  experts.  The  tryout  will  be  conducted  one- 
on-one  to  confirm  that  the  masters  can  consistently  pass  the  Performance 
Measures  or  items  and  to  confirm  the  acceptability  of  the  Scoreable  Unit 
as  a measure  of  ability  to  perform  the  task.  Different  groups  of  experts 
may  need  to  be  assembled  to  cover  all  Scoreable  Units.  In  addition  to 
the  tryout  on  five  experts,  the  hands-on  Scoreable  Units  will  be  tried 
out  on  five  novices  with  four  raters  to  determine  scoring  agreement  of 
the  raters.  Satisfactory  Performance  Measures  can  be  scored  consistently 
by  the  raters;  unsatisfactory  Performance  Measures,  which  cannot  be 
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scored  consistently,  must  be  revised  until  raters  can  agree  on  their 
scoring.  Following  this  try  out  of  all  Scoreable  Units,  the  entire 
components  will  be  submitted  to  ITEG,  together  with  (1)  the  task  analy- 
sis for  each  task  tested,  (2)  a summary  of  the  results  of  the  tryout  and 
review,  (3)  all  administrative  instructions,  and  (4)  the  SQT  Notice. 

The  task  analysis  will  have  the  essential  and  key  steps  identified.  ' 

ITEG  will  review  the  components  to  judge  the  adequacy  of  the  Scoreable  | 

Units  to  measure  performance  on  the  essential  and  key  steps  of  the  , 

tasks.  This  step  corresponds  to  Phase  I in  Inclosure  1.  j 

d.  The  content  of  the  Scoreable  Units  cannot  be  changed  after  ITEG  t 

approval  (except  by  approval  from  ITEG).  Items  or  Performance  Measures 
found  to  be  unsatisfactory  on  the  basis  of  tryout  on  soldiers  will  be 
revised  to  improve  them,  but  not  to  the  extent  of  changing  content. 

Specifically,  unsatisfactory  items  or  Performance  Measures  cannot  be 
deleted  without  approval.  They  will  be  revised  until  satisfactory. 

e.  The  final  ITEG  review  is  of  the  complete  SQT  after  it  has  been 
tried  out  on  soldiers;  a summary  of  the  tryout  results  must  also  be 
submitted  to  ITEG.  The  entire  SQT  package  is  submitted  to  ITEG  for 
review  and  approval  prior  to  operational  administration. 

f.  In  the  November  1975  validation  procedures,  the  written  and 
hands-on  components  were  scheduled  for  submission  to  ITEG  after  tryout 
with  representative  soldiers  in  the  field  (called  Phase  II  in  that 
document).  The  content  review  will  no  longer  be  delayed  until  after 
Phase  II  (as  detailed  in  para  3b  and  c) . The  final  step  of  Phase  II  is 
now  the  submission  of  validated  written  and  hands-on  components  and 
summary  of  tryout  results.  The  review  procedures  described  above 
supercede  the  procedure  contained  in  the  November  1975  document. 

g.  The  primary  reason  for  early  ITEG  review  of  the  Scoreable  Units 
is  to  provide  timely  guidance  to  the  TDA  about  the  adeiuacy  of  the 
procedures  for  developing  Scoreable  units.  The  old  procedures  of  ITEG 
review  after  tryout  on  soldiers  occurred  too  late  to  be  of  much  use  in 
the  development  process.  Under  the  new  review  procedures,  the  TDA  are 
assured  that  the  final  SQT  submitted  to  ITEG  after  tryout  will  be 
acceptable  for  operational  administration. 

4.  The  validation  procedures  have  been  improved  to  make  them  more 
useful  for  low-density  MOS,  as  well  as  high-density  MOS.  The  details  of 
how  to  conduct  the  validation  are  presented  in  Inclosure  2.  The  hands- 
on  component,  after  approval  of  the  test  by  ITEG,  will  be  tried  out  in  a 
representative  unit.  The  written  component,  again  after  approval  of  the 
content  by  ITEG,  will  be  tried  out  on  30  soldiers.  The  tryout  can  be 
conducted  locally  on  soldiers  that  are  available  or  in  field  units  on 
representative  soldiers,  or  on  a combination  of  locations  and  soldiers. 

Some  special  considerations  that  apply  to  the  written  component  are  as 
follows: 
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a.  Written  Scoreable  Units  will  be  evaluated  on  two  bases.  The 
first  is  that  performance  on  the  items  must  be  related  to  self-ratings 
of  ability  to  perform  the  task  covered  by  the  Scoreable  Units.  The 
self-ratings  are  used  to  identify  groups  of  experts  and  novices,  and  the 
grouping  of  the  sample  into  experts  and  novices  may  be  different  for 
each  task.  Most  soldiers,  no  matter  how  experienced,  are  not  experts  on 
all  tasks  Included  in  an  SQT.  The  second  basis  for  evaluating  items  is 
that  all  of  them  in  a Scoreable  Unit  must  be  consistent  indicators  of 
ability  to  perform  the  task.  In  other  words,  the  degree  of  consistency 
is  measured  by  how  well  each  pair  of  items  agrees  in  indicating  mastery/ 
nonmastery.  Soldiers  who  can  perform  the  task  should  tend  to  pass  the 
items,  and  soldiers  who  cannot  perform  the  task  should  tend  to  fail  the 
items.  Items  that  are  unsatisfactory  on  one  or  both  bases  must  be 
revised  until  they  are  satisfactory.  When  revising  the  unsatisfactory 
items,  care  must  be  exercised  not  to  change  the  test  content. 

b.  The  reason  for  evaluating  the  written  Scoreable  Units  against 
both  self-ratings  and  Item-Scoreable  Unit  agreement  is  to  obtain  as  much 
information  as  possible  about  the  quality  of  the  items  from  the  tryout. 
Locating  soldiers  to  take  the  test  and  administering  the  test  is  time 
consuming,  and  therefore  should  be  held  to  a minimum.  Items  that  are 
satisfactory  on  both  bases  are  more  likely  to  be  satisfactory  in  opera- 
tional administration.  Items  unsatisfactory  on  both  counts  require 
revision,  and  items  unsatisfactory  on  one  of  the  bases  should  be  exam- 
ined carefully  to  see  if  revision  is  required.  All  items  will  be  evalua- 
ted again  by  ITEG  after  operational  administration  but  prior  to  setting 
the  final  scoring  key,  and  all  unsatisfactory  items  will  be  deleted 
before  setting  the  final  scoring  key.  However,  if  unsatisfactory  items 
are  administered  operationally,  soldiers  waste  precious  testing  time 
taking  these  items  and  the  effectiveness  of  the  Scoreable  Units  is 
weakened . 

c.  The  entire  written  component  will  be  tried  out  at  the  same  time. 
This  way  the  testing  time  can  be  estimated.  Testing  time  is  a scarce 
resource;  therefore,  TDA  should  take  full  advantage  of  the  allotted  four 
hours,  with  the  only  provision  being  that  90  percent  of  the  soldiers 
must  finish  the  component  within  four  hours.  Scoreable  Units  found 
satisfactory  during  the  first  tryout  need  not  be  evaluated  again. 
Unsatisfactory  items  and  Scoreable  Units  must  be  revised  and  tried  out 
until  they  are  satisfactory.  Changes  to  the  test  content  must  be 
approved  by  ITEG. 

d.  Written  components  that  have  special  administrative  requirements 
and  instructions  must  be  tried  out  by  a representative  TCO  at  a repre- 
sentative installation  to  determine  adminstrative  feasibility.  The 
response  of  the  soldiers  from  this  administration  can  be  included  with 
the  results  of  the  previous  tryout  to  evaluate  the  Scoreable  Units.  The 
administrative  requirements  and  instructions,  if  not  satisfactory,  must 
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be  revised  until  they  are  satisfactory. 

e.  The  reason  for  trying  out  the  Scoreable  Units  prior  to  opera- 
tional administration  is  to  Improve  their  quality  as  measures  of  ability 
to  perform  the  tasks.  The  sample  of  soldiers  used  in  the  tryouts  should 
be  familiar  with  tasks  covered  by  the  Scoreable  Units,  and  the  range  of 
ability  in  the  sample  should  be  from  the  experienced  expert  to  the 
beginning  novice.  If  the  sample,  as  a whole,  lacks  familiarity  with  the 
task  covered  by  a Scoreable  Unit,  then  the  results  of  the  tryout  are  of 
little  value.  If  most  soldiers  respond  randomly  to  the  items,  then  the 
Scoreable  Unit  may  appear  to  be  unsatisfactory  but  in  fact  it  may  be 
satisfactory  if  tried  out  on  experienced  soldiers.  Therefore,  if  a 
Scoreable  Unit  is  found  to  be  unsatisfactory  and  expert  review  shows  no 
faults  in  the  items  and  the  sample  contains  few  self-rated  experts,  then 
the  Scoreable  Unit  should  be  tried  out  again  on  another  group  of  soldiers 
who  are  experienced  on  the  task.  As  a rule,  experienced  soldiers  provide 
more  useful  information  about  Item-Scoreable  Unit  agreement,  and  these 
should  be  included  in  the  sample.  Some  novices  should  also  be  included 
to  make  sure  that  the  Scoreable  Units  do  Indeed  distinguish  between 
masters  and  nonmasters. 

f.  The  sample  size  should  be  a minimum  of  30  soldiers  to  evaluate 
Scoreable  Units.  With  samples  of  this  size,  individual  responses  hav^  a 
relatively  small  effect  on  the  trend  for  the  group.  In  many  cases,  30 
soldiers  are  not  available  for  tryout  of  the  written  component.  In  such 
a case,  obtain  as  many  experienced  soldiers  as  possible,  including  those 
in  related  MOS  who  may  have  some  familiarity  with  the  tasks,  and  admin- 
ister the  test  to  them.  Since  the  SQT  will  be  evaluated  after  opera- 
tional admlnstration  and  unsatisfactory  items  and  Scoreable  Units 
deleted,  the  likelihood  of  fielding  a satisfactory  SQT  is  Increased  by 
trying  out  the  Scoreable  Units  on  a group  as  representative  as  possible 
of  the  target  population. 

5.  Representative  soldiers  in  the  field  are  not  readily  available  for 
many  of  the  MOS.  A problem  that  occurs  especially  in  many  machine- 
oriented  MOS  is  that  soldiers  in  a unit  tend  to  be  specialized.  In  a 
tryout  of  the  SQT  on  these  soldiers,  they  can  provide  useful  information 
for  the  Scoreable  Units  releva:.t  to  their  assigned  job  tasks,  but  only 
limited  information  on  the  remaining  Scoreable  Units.  Since  the  Soldier's 
Manuals  and  SQT  Notice  are  usually  not  available  in  sufficient  time  for 
study,  the  soldiers  have  no  opportunity  to  prepare  for  the  test,  and  the 
results  are  not  truly  representative  of  those  expected  in  operational 
administration.  The  new  validation  procedures,  therefore,  delete  the 
requirement  that  the  SQT  be  evaluated  in  two  phases.  Inclosure  1 speci- 
fies that  both  the  hands-on  and  written  components  be  tried  out  on 
soldiers  in  only  one  phase,  now  called  Phase  II.  Phase  III  validation 
is  part  of  the  operational  admlnstration  process.  A representative 
sample  of  soldiers'  tests  are  selected  and  used  for  the  item  analysis  of 
Scoreable  Units  prior  to  final  scoring  and  grading. 


OUTLINE  OF  VALIDATIC»)  PROCEDURE 


Phase  0 - SELECT  TASKS 

• Develop  SQT  Plan 

. List  tasks  to  be  tested 

. Allocate  tasks  to  component: 

Hands-on 

Written 

- Performance  Certification 
. Rationale 

. Soldier's  Manual 

Submit  to  ITEG  for  Review  and  Approval 

• Construct  Sample  of  Scoreable  Units 

. Task  Analyses,  specify  Essential  and  Key  steps 

. Scoreable  Units 

5-6  Written 
2 Hands-on 

All  Performance  Certification 
. SQT  Notice 

Submit  to  ITEG  for  Review  and  Approval 

• TASKS  ARE  ESTABLISHED 


vit 


Phase  I - ESTABLISH  SPECIFIC  CONTENT 
HANDS-ON  AND  WRITTEN  COMPONENTS 

• Test  all  Scoreable  Units  on  5 Experts;  revise  as  required 

. One-on-one  Tryout 

. Confirm  Mastery 

. Determine  Acceptability 

• Try  Out  Hands-on  Scoreable  Units  on  5 Novices  with  4 Raters; 
revise  as  required 

. Compute  Rater  Agreement 

• Assemble  Components 

. Task  Analysis,  specify  Essential  and  Key  Steps 

. Summary  of  Tryout 

. Administrative  Instructions 

. SQT  Notice 

Submit  to  ITEG  for  Review  and  Approval 

• SCOREABLE  UNITS  ARE  ESTABLISHED 
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Phase  II  - ESTABLISH  VALIDITY  OF  COMPONENTS 
HANDS-ON  COMPONENT 
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• I'ry  Out  Locally  on  Representative  Soldiers  with  4 Raters; 
revise  as  required 

. Compute  Rater  Agreement 

. Determine  Acceptability 

Soldiers 

Raters 

. Judge  Administrative  Feasibility 

• Try  Out  with  Representative  TCO,  Soldiers,  and  Raters;  revise 
as  required 

. Determine  Administrative  Feasibility 

. Determine  Acceptability 

Soldiers 
Raters 

• Assemble  Final  Component 

. Summary  of  both  Tryouts 

Submit  to  ITEG  for  Review  and  Approval 
WRITTEN  COMPONENT 

• Try  Out  2 Scoreable  Units  Against  Performance  Tests 

. Confirm  Mastery 

. Compute  Item-Performance  Test  Agreement 

• Try  Out  on  30  Soldiers;  revise  as  required 

. Identify  Experts  and  Novices  with  Self-Ratings 

. Compute  Self-Rating  - Scoreable  Unit  Agreement 
. Compute  Item  - Scoreable  Unit  Agreement 

. Determine  Acceptability 

a Try  Out  Component  Containing  Unique  Instructions  with 
Representative  TCO  and  Soldiers;  revise  as  required 

. Determine  Administrative  Feasibility 

• Assemble  Final  Component 

. Summary  of  all  Tryouts 


Submit  to  ITEG  for  Review  and  Approval 
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PHASE  0 - SELECT  TASKS 
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1.  Develop  SOT  Plan. 

a.  The  SQT  Plan  contains  the  following  information: 

(1)  list  of  tasks  to  be  tested 

(2)  which  component  (hands-on,  written,  or  performance 
certification)  a task  is  to  be  tested  in 

(3)  a rationale  for  the  selection  and  allocation  approaches 

(4)  a draft  Soldier's  Manual,  the  Soldier's  Manual  pages 
corresponding  to  each  task  Included  in  the  SQT  plan, 
or  if  the  Soldier's  Manual  is  not  available,  a complex 
task  analysis  for  each  selected  task. 

b.  Select  tasks  for  testing.  When  selecting  tasks,  place  emphasis 

on  task  Importance  rather  than  random  coverage  across  all  functional 
areas.  The  two  major  classes  of  importance  are:  (1)  criticality  to 

mission  accomplishment,  based  on  expert  judgments,  and  (2)  performance 
deficiencies  in  the  field,  documented  by  field  data  demonstrating  weak 
performance.  Potential  sources  of  data  include  ARTEP  results,  main- 
tenance management  center  data,  ESC  report,  IG  Inspection  reports,  and 
morning  report. 

c.  Assign  tasks  to  components.  Whenever  possible,  assign  tasks  to  the 
component  in  which  they  most  naturally  fit.  Since  for  most  tasks,  performance 
requires  physical  skills,  which  are  best  measured  in  the  hands-on  component, 
most  tasks  should  be  tested  in  the  hands-on  component  whenever  possible. 
Feasibility,  task  content,  task  criticality,  and  specification  of  key  and 
essential  steps  are  among  the  major  allocation  criteria.  (Essential  steps  are 
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behaviors  necessary  to  perform  the  task,  and  key  steps  are  the  behaviors  that 
have  serious  consequences  to  personnel  or  equipment,  or  are  frequent  sources 
of  failure  In  performing  the  task.  Criteria  for  allocating  tasks  to  components 
are  listed  In  a separate  document. 

d.  Submit  SOT  plan  to  ITEG  for  review  and  approval.  ITEG  will  complete 
their  review  of  the  SQT  plans  within  10  working  days  after  receiving  it. 

e.  Revise  SQT  plan  as  required  bv  ITEG. 
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2.  Construct  a satisfactory  SQT  sample. 

a.  Construct  a sample  of  Scoreable  Units.  The  sample  is  to  consist  of 

(1)  5 or  6 written  Scoreable  Units 

(2)  2 hands-on  Scoreable  Units 

(3)  all  performance  certification  Scoreable  Units 

(4)  SQT  Notice  corresponding  to  the  submitted  Scoreable  Units 

(5)  Task  analyses  corresponding  to  the  submitted  Scoreable  Units. 

The  task  analyses  shall  identify  all  essential  and  key 
behaviors. 

b.  Submit  the  sample  of  Scoreable  Units  to  ITEG  for  review  and 
approval.  ITEG  will  review  the  content  of  Scoreable  Units  to  judge  whether 
the  content  of  the  Scoreable  Units  adequately  covers  actual  task  performance. 
ITEG  will  reply  to  the  Test  Development  Activity  within  10  working  days  after 
receiving  the  sample  Scoreable  Units. 

c.  If  ITEG  approves  the  sample,  i.e.,  finds  the  content  of  the  sample 
satisfactory,  develop  the  entire  written  and  hands-on  components  and  proceed 


to  Phase  1. 

d.  If  ITEG  does  not  approve  the  sample,  that  is,  finds  the  content  of  the 
sample  unsatisfactory,  revise  the  sample  and  also  construct  a second  sample 


of  Scoreable  Units 


e.  Submit  the  second  sample  of  Scoreable  Units  to  ITEG 


* I f«  If  ITEG  approves  the  second  sample,  develop  the  entire  written  and 
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t hands-on  components  and  proceed  to  Phase  1. 

\ ' 

^ j g.  If  ITEG  does  not  approve  the  second  sample,  direct  contact  between 

I : ITEG  and  Test  Development  Activity  (TDA)  representatives  Is  required.  Once 

I ^ ITEG  has  approved  a sample  of  Scoreable  Units  produced  by  the  TDA,  the  TDA 

f’ 

I will  develop  the  entire  written  and  hands-on  components  and  proceed  to 

r 

I Phase  I. 

r 

t 


PHASE  I - ESTABLISH  SPECIFIC  CONTENT 


Hands-on  Component 

Description  of  Hands-on  Component:  Each  Scoreable  Unit  In  the  hands-on 

component  contains  all  the  Information  needed  to  administer  and  score  the 
Scoreable  Unit.  This  Includes:  (a)  administrative  Instructions  de- 

S 

tailing  how  to  set  up  the  test  stations  for  their  Scoreable  Units,  (b) 
pretest  conditions  checklists  Indicating  the  status  of  equipment  and 
materials  before  any  soldier  Is  tested,  (c)  specific  testing  Instructions 
stating  exactly  what  is  read  to  the  soldier  and  what  the  soldier  reads 
to  himself  in  case  printed  instructions  are  used,  and  (d)  scoring 
instructions  listing  what  Performance  Measures  are  scored  and  the  exact 
manner  in  which  each  Performance  Measure  is  scored.  Include  a pretest 
conditions  checklist,  testing  instructions,  and  scoring  instructions  on 
a one-page  scoresheet  which  can  be  used  for  scoring  each  soldier  after 
setting  up  the  test  station. 

1.  Administer  Scoreable  Units  to  available  test  developers  one  at  a time. 
These  informal  administrations  may  Identify  Performance  Measures  and 
Scoreable  Units  in  need  of  major  revision. 

2.  Test  all  Scoreable  Units  with  at  least  five  experts.  Conduct  one-on-one 
tryout  locally  on  experts  to  confirm  that  the  masters  of  a task  can 
consistently  pass  the  Performance  Measures  of  the  Scoreable  Unit  derived 
from  the  task  and  to  confirm  the  acceptability  of  the  Scoreable  Unit  as 

a measure  of  ability  to  perform  the  task.  Different  groups  of  experts  may 
need  to  be  assembled  to  cover  all  Scoreable  Units. 
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a.  Select  experts  by  first  providing  soldiers  with  a detailed  des- 
cription of  the  task  that  specifies  the  task  conditions  and  standards, 
and  then  asking  them  to  answer  a self-rating  question.  Two  prototype 
self-rating  formats  are  provided.  Select  the  question  most  appropriate 
for  the  set  of  tasks,  and  have  the  soldiers  answer  that  question. 

How  well  can  you  perform  the  task? 

0.  Not  familiar  with  the  task. 

1.  Not  very  well. 

2.  Fairly  well. 

3.  Very  well. 

To  perform  the  complete  task,  how  much  technical  guidance 

would  you  require  from  someone  who  knows  how  to  perform  the 

task? 

0.  Not  familiar  with  the  task 

1.  Lots. 

2.  Some. 

3.  None. 

For  validating  hands-on  Scoreable  Units,  experts  are  soldiers  who  select 
alternative  3,  "Very  well,"  or  "None." 

b.  Test  one  expert  at  a time  and  have  each  expert  scored  by  two 
raters  (one  to  be  the  test  administrator  and  one  to  be  the  observer) . The 
raters  must  score  Independently  of  each  other.  If  the  two  raters  do  not 
agree  on  each  Performance  Measure,  then  the  scoresheet  or  administrative 
instructions,  or  both,  must  be  revised  before  testing  another  expert. 
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c.  Test  anoLner  expert  on  the  Scoreable  Unit;  If  the  examiners 
agree  on  all  Performance  Measures,  then  continue  testing  experts  until 
perfect  agreement  between  the  two  examiners  Is  obtained  on  five  experts.  ' 

d.  Require  that  at  least  80  percent  of  the  experts  be  scored  GO  on  ‘ 

the  Scoreable  Unit  and  at  that  at  least  80  percent  pass  each  Performance 
Measure  within  the  Scoreable  Unit.  (A  summary  form  to  be  used  for  ITEG 
submittal  Is  located  on  p.  51  .) 

e.  Determine  acceptability  of  the  Scoreable  Unit  to  the  experts 
Immediately  after  they  are  tested.  Determine  the  criticality,  fairness, 
and  feasibility  of  the  Scoreable  Unit  by  having  the  experts  review  the  task 
analysis  and  the  specification  of  key  and  essential  behaviors,  and  then 
complete  question  sheets  l and  2 (pp.  44-45).  Require  that  at  least  80  percent 
of  the  experts  answer  yes  to  each  question.  (A  summary  form  to  be  used 

for  ITEG  submittal  is  located  on  p.  49  .) 

f.  Revise  Performance  Measure(s)  and  Scoreable  Units  as  required. 

3.  Try  out  all  Scoreable  Units  on  at  least  5 novices  with  4 raters  to 
determine  rater  agreement.  Satisfactory  Performance  Measures  can  be 
scored  consistently  by  the  raters;  unsatisfactory  Performance  Measures,  ' ' 
cannot  be  scored  consistently. 

a.  Select  groups  of  novices  by  first  providing  soldiers  with  a 
detailed  description  of  the  task  that  specifies  the  task  conditions  and 
standards,  and  then  asking  them  one  of  the  questions  In  la,  above  (self- 
ratings  of  ability  to  perform  the  tasks) . Novices  are  soldiers  who  select 
alternative  1,  "Not  very  well"  or  "Lots". 
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b.  Test  one  novice  at  a time  and  have  each  novice  scored  by  four 


t-i 

li 

P 

f 1 


raters  (one  serving  as  the  test  administrator).  The  raters  must  score 
Independently  of  each  other. 

c.  Determine  the  scoring  agreement  between  all  possible  pairs  of 
raters  for  each  Performance  Measure.  At  least  80  percent  of  the  rater 
pairs  must  agree  on  each  Performance  Measure,  for  the  Performance  Measure 
to  be  satisfactory. 

Consider  the  following  example  of  a Performance  Measure  In  a hands-on 
Scoreable  Unit.  In  this  example  there  are  five  novices  and  each  was 
scored  pass  or  fall  by  all  four  raters.  The  following  codes  are  used 
In  the  table: 

A ~ agreements  among  pairs  of  raters 
D ■ disagreements  among  pairs  of  raters 
1 « novice  scored  pass  on  the  Performance  Measure 
0 » novice  scored  fall  on  the  Performance  Measure 

EXAMPLE  OF  A SATISFACTORY  PERFORMANCE  MEASURE 


NOVICES 


1 

2 

3 

4 

5 


RATERS 

1 2 3 4 A D 

11116  0 
11116  0 
10  113  3 

110  13  3 

0 0 0 0 6 0 


24  6 

Construction  and  Interpretation  of  Table  to  Compute  A.greement 

(1)  List  the  novices  and  raters  as  shown  above. 

(2)  Record  the  pass  or  fall  score  assigned  to  each  novice  by  each 
of  the  raters. 
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(3)  Record  the  number  of  agreements  (A)  and  disagreements  (D)  between 
pairs  of  raters.  For  example,  novices  1 and  2 were  scored  pass  by  all 
raters,  which  means  that  all  six  pairs  of  raters  were  In  agreement: 

1&2,  1&3,  1&4,  2&3,  2&4,  and  3 & 4.  Record  a 6 under  column  A 
and  0 under  column  D for  novices  1 and  2.  Novice  3 was  scored  pass  by 
three  raters  and  fall  by  one  rater,  which  means  that  three  pairs  of 
raters  were  In  agreement  (1  & 3,  1 & 4,  and  3 & 4)  and  three  pairs  were  In 
disagreement  (1  & 2,  2 & 3,  and  2 & 4) . 

(4)  Add  the  nimbers  In  the  A and  D columns.  In  this  example  there 
were  24  In  the  A column  and  6 In  the  D column. 

(5)  Calculate  the  agreement  between  rater  pairs.  In  this  example: 

Percent  Agreement  between  rater  pairs  • [24  4 (24  + 6)]x  100  • 80%; 
therefore,  the  80  percent  agreement  criterion  Is  satisfied. 

(6)  Repeat  this  procedure  for  each  Performance  Measure  of  a Scoreable 
Unit. 

(7)  Identify  all  Performance  Measures  with  less  than  80  percent 
rater  agreement.  Indicate  which  Performance  Measures  are  unsatisfactory 

with  respect  to  rater  agreement  on  the  summary  forms  submitted  to  ITEG.  (p.  52  ) 

ITEG  will  consider  this  Information  In  Its  evaluation  of  the  content  and 
feasibility  of  the  Hands-on  Scoreable  Units. 

EXAMPLE  OF  AN  UNSATISFACTORY  PERFORMANCE  MEASURE 
NOVICES  RATERS 

1 2 3 4 A D 

1 11116  0 

2 110  13  3 

3 10  0 12  4 

4 1110  3 3 

5 0 1 1 0 2 4 

16  14 
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% Agrecm*:..,  batWMn  rater  pairs  ■ [16  * (16  14)]  x 100  “ 53Z 

In  this  example 1 the  egreamant  was  leas  than  the  required  80  percent. 

d.  Determine  acceptability  to  raters.  The  raters  of  each  Scoreeble 

Unit  must  complete  question  sheet  2;  seventy-five  percent  must  Indicate 

* 

acceptance  of  each  Scoreable  Unit  as  valid,  fair,  and  feasible.  (A  sumsisry 
form  to  be  used  for  ITEG  submittal  Is  located  on  p.  50.) 

3.  Assemble  the  hands-on  component  to  Include: 

(a)  the  Scoreable  Units  of  the  hands-on  component; 

(b)  summary  of  the  tryouts  on  experts  and  novices  (use  the  summary  forms); 

(c)  task  analysis  specifying  the  key  and  essential  steps  for  each  task; 

(d)  all  administrative  instructions; 

(e)  the  SQT  Notice  (identifying  that  portion  specific  to  the  hands-on 
component) . 

5.  Submit  the  hands-on  component  and  all  other  requested  documentation  to 
ITEG  for  review  and  approval. 

6.  Revise  the  hands-on  component  as  requested  by  ITEG.  Obtain  final  appro- 
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PHASE  I - ESTABLISH  SPECIFIC  CONTENT 
Written  Component 

1.  Administer  Scoreable  Units  to  available  test  developers  one  at  a time. 

These  Informal  administrations  will  Identify  Items  and  Scoreable  Units  In  need  of 
major  revision.  Revise  Scoreable  Units  In  accordance  with  recommenda- 
tions of  the  test  developers.  (A  prototype  form  for  this  step  is  provided 

on  p.  63  .) 

2.  Test  all  Scoreable  Units  with  five  experts.  Conduct  one-on-one  tryout 
on  experts  to  confirm  that  the  masters  of  a task  can  consistently  pass 
the  Items  of  a Scoreable  Unit  derived  from  the  task  and  to  confirm  the 
acceptability  of  the  Scoreable  Unit  as  a measure  of  ability  to  perform 
the  task.  Different  groups  of  experts  may  need  to  be  assembled  to  cover 
all  Scoreable  Units. 

a.  Select  experts  by  first  providing  soldiers  with  a detailed 
description  of  the  task  that  specifies  the  task  conditions  and  standards, 
and  then  asking  them  to  answer  a self-rating  question.  Two  prototype 
self-rating  forms  are  provided.  Select  the  question  most  appropriate 
for  the  set  of  tasks  and  have  the  soldiers  answer  that  question. 

How  well  can  you  perform  the  task? 

0.  Not  familiar  with  the  task. 

1.  Not  very  well. 

2.  Fairly  well. 

3.  Very  well. 

To  perform  the  complete  task,  how  much  technical  guidance 
would  you  require  from  someone  who  knows  how  to  perform  the  task? 

0.  Not  familiar  with  the  task. 

1.  Lots. 

2.  Some. 

3.  None. 


For  validating  written  Scoreable  Units,  experts  are  soldiers  who  select 
alternative  3,  "Very  well,"  or  "None." 


b.  Test  one  expert  at  a time.  Each  expert  Is  tested  on  the  Scoreable 


Unit  to  confirm  the  adequacy  of  the  scoring  key.  After  each  expert 
evaluates  the  Scoreable  Unit  necessary  revisions  are  made  to  the  Scoreable 
Unit  prior  to  tryout  on  the  next  expert. 

c.  Require  that  at  least  80  percent  of  the  experts  agree  on  the 
scoring  key.  (A  summary  form  to  be  used  for  ITEG  submittal  Is  provided 
on  p.  54,  a,  b.) 

d.  Determine  acceptability  of  the  Scoreable  Unit  to  the  experts. 

Determine  the  criticality,  fairness,  and  completeness  of  the  Scoreable  Unit 
by  having  the  experts  review  the  task  analysis  and  the  specification  of  key 

and  essential  behaviors,  and  complete  question  sheet  1 and  3 (pp.  44  & 46).  Require 
that  at  least  80  percent  of  the  five  experts  answer  yes  to  each  question. 

(A  summary  form  to  be  used  for  ITEG  submittal  is  located  on  p.  53,  a,  b.) 

e.  Revise  item(s)  and/or  Scoreable  Units  as  required. 

3.  Assemble  the  written  component  to  include: 

(a)  the  Scoreable  Units  of  the  written  component; 

(b)  summary  of  the  tryouts  on  experts  (use  the  summary  submittal 
forms) ; 


(c)  task  analysis  specifying  the  key  and  essential  steps  for  each 


task; 

(d)  all  administrative  instructions; 

(e)  the  SQT  Notice  (identifying  that  portion  specific  to  the  written 
component) 

4.  Submit  the  written  component  and  all  other  requested  documentation  to 
ITEG  for  review  and  approval. 

5.  Revise  the  written  component  as  requested  by  ITEG.  Obtain  final 


approval  from  ITEG* 


1 


Once  the  written  component  has  been  approved  by 
ITEG  the  content  of  each  item  within  each 


Scoreable  Unit  Is  fixed  and  cannot  be  changed 
without  approval  from  ITEG.  Items  found  to 
be  unsatisfactory  during  Phase  II  Validation 
will  be  revised  to  Improve  them,  but  not  to 
the  extent  of  changing  content.  Unsatisfactory 
Items  cannot  be  deleted  without  approval.  


PHASE  II  - ESTABLISH  VALIDITY 


Hands-on  Component 

I 

i Try  out  each  Scoreable  Unit  locally  on  10  representative  soldiers  with 

! 

4 raters.  The  raters  must  be  adequately  and  uniformly  prepared  to  perform 
their  function.  A session  where  the  raters  practice  administering  and 
scoring  their  Scoreable  Unit  will  help  prepare  them  to  score  more  accurately. 
Each  Scoreable  Unit  must  be  administered  to  at  least  10  soldiers,  and  the 
performance  of  each  soldier  must  be  scored  Independently  by  an  administrator 
and  three  other  raters. 

a. Compute  the  agreement  between  pairs  of  raters  for  the  Performance 
I Measures  of  the  Scoreable  Units.  Require  at  least  80  percent  agreement  among 

raters  for  each  Performance  Measure.  Detailed  discussion  of  how  to  compute 
rater  agreement  Is  provided  In  the  PHASE  I.  Hands-on  Component  section. 

The  following  Is  a specific  example  of  a case  In  which  sufficient  agreement 
Is  obtained  (at  least  80  percent  agreement)  and  a case  where  sufficient 
agreement  Is  not  obtained  (less  than  80  percent  agreement) . (A  summary 
form  to  be  used  for  ITEG  submittal  Is  provided  on  p.  57  ). 

1 

? 

I 

i 

i 


SOLDIERS 


RiVTERS 


ADM  1 


50  10 


Z Agreement  between  pairs  of  raters  “ {50  (50  + 10)}  x 100  = 83Z 

Therefore,  there  is  sufficient  agreement  on  this  Performance  Measure* 


RATERS 


EXAMINEES 


ADM  1 


37  23 

Z Agreement  between  pairs  of  Raters  = (37  + (37  + 23)}  x 100  = 61Z 
Therefore,  there  is  not  sufficient  agreement  on  this  Performance  Measure > 
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b.  Deterralne  acceptability  to  soldiers  and  raters.  The  10  soldiers 


must  complete  question  sheet  4 (p.  47  );  eighty  percent  must  Indicate 

acceptance  of  each  Scoreable  Unit  as  a fair  test  of  their  ability.  The 
4 raters  of  each  Scoreable  Unit  must  complete  question  sheet  2;  seventy-five 
percent  must  Indicate  acceptance  of  each  Scoreable  Unit  as  valid,  fair, 
and  feasible.  (Summary  forms  to  be  used  for  ITEG  submittal  are  provided 
on  p. 55-56.) 

c.  Revise  Performance  Measure(s)  and/or  Scoreable  Unit(s)  that  do 
not  satisfy  the  rater  agreement  criterion  or  the  acceptability  criterion. 

The  revision  cannot  change  the  content  of  a Performance  Measure  or  Scoreable 
Unit  unless  ITEG  approves. 

d.  Retest  on  10  representative  soldiers  with  4 raters  any  Scoreable 
Unit  that  was  found  unsatisfactory  and  required  revision.  Continue  to  retest 
until  both  rater  agreement  and  acceptability  criteria  are  satisfied. 

e.  Judge  administrative  feasibility  of  the  entire  hands-on  component 
on  the  basis  of  this  tryout;  acceptability  question  sheets  may  provide 
Indications  of  feasibility. 

2.  Try  out  hands-on  component  with  representative  TCP,  soldiers  and  raters 
to  determine  administrative  feasibility. 

a.  Select  an  appropriate  unit  from  FORSCOM  and  Army  Reserve  or  National 
Guard.  These  units  will  be  selected  Jointly  by  the  TDA  and  ITEG  through 
the  major  command.  The  units  selected  should  be  those  having  the  maximum 
number  of  soldiers  available  for  testing. 

b.  Send  a package  to  the  TCP  that  contains  all  necessary  materials 
Including  Instructions  for  setting  up  the  test  site  and  conducting  the 
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I 


J 

'■i 

i 
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( 
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hands-on  component.  The  package  should  contain  enough  materials  to  allow 
testing  the  maximum  number  of  soldiers  in  two  sessions.  One  session  should 
be  conducted  in  the  morning,  and  if  necessary,  another  session  should  be 
conducted  in  the  afternoon.  Moreover,  the  TDA  must  send  an  observer  to 
determine  administrative  feasibility  of  the  hands-on  component  and  to 
determine  how  many  soldiers  can  be  tested  per  hour  for  given  amounts  of 
resources  (number  of  examiners  and  amounts  of  equipment) . 

c.  Administer  the  hands-on  component  to  at  least  10,  and  preferably 
up  to  50,  representative  soldiers  to  determine  administrative  feasibility 
under  operational  tasting  conditions. 

d.  Conduct  after-action  interview  of  unit  personnel  to  include  test 
administrator,  raters,  soldiers  and  any  other  appropriate  individuals. 

e.  Assess  acceptability  to  unit  soldiers  and  raters.  Use  question 
sheets  2 and  4 during  this  step. (Summary  forms  to  be  used  for  ITEG 
submittal  are  located  on  pp.  55-56.) 

f.  Revise  hands-on  component  as  required  to  insure  administrative 
feasibility. 

3.  Assemble  hands-on  component  to  include  a summary  of  tryout  results. 

4.  Submit  hands-on  component  to  ITEG  for  review  and  approval. 

5.  Revise  the  hands-on  component  in  accordance  with  ITEG  instructions. 

6.  Assemble  final  hands-on  component. 

7.  Submit  final  hands-on  component  to  ITEG  for  operational  administration. 


I 


I 

f 
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PHASE  II  - ESTABLISH  VALIDITY 

Written  Component  j 

1.  Validate  at  least  two  written  Scoreable  Units  against  a performance  | 

test  of  the  corresponding  tasks.  Comparing  Scoreable  Unit  and  performance  I 

test  performance  will  indicate  the  degree  to  which  the  written  Scoreable  j 

I 

Units  are  valid  measures  of  task  performance.  CA  two-page  form  to  be  j 

used  for  ITEG  submittal  is  provided  on  p.  58, a .) 

a.  Two  or  more  experts  develop  administrative  and  scoring  procedures  j 

by  administering  the  performance  test  to  each  other  or  another  individual.  | 

b.  Administer  both  the  performance  test  and  the  set  of  written  items 

\ 

based  on  the  task  to  at  least  five  soldiers  who  are  expected  to  be  experts  | 

1 

and  five  who  are  expected  to  be  novices. 

c.  Require  that  at  least  80  percent  of  the  expert  group  pass  the 
performance  test,  and  require  that  a greater  proportion  of  the  experts 
than  the  novices  pass  the  performance  test. 

d.  Proceed  to  1 e.if  the  requirements  stated  in  1 c.are  satisfied. 

If  the  requirements  in  1 c.  are  not  satisfied  either  revise  the  written 
items,  select  new  expert  and  novice  groups,  or  both.  Recall  that  any 
change  to  the  items  in  a Scoreable  Unit  cannot  change  its  original  content 
without  approval  from  ITEG.  Continue  the  process  until  at  least  80  percent  of 
the  expert  group  pass  the  performance  test  and  a greater  proportion  of 
experts  than  novices  pass  the  performance  test. 
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e.  Determine  whether  or  not  agreement  between  the  performance  test 
and  each  of  the  written  items  is  satisfactory.  For  each  item  in  the 
Scoreable  Unit,  prepare  a table  as  shown  below.  An  Agreement  Index  is 
then  calculated  between  the  performance  test  and  each  written  item. 


PERFORMANCE  TEST 


ITEM  Pass Fail 


a = Number  of  s.;ldiers  who 

b = Number  of  soldiers  who 

Pass 

Pass  performance  test  and 

Fail  performance  test  and 

Pass  written  item 

Pass  written  item 

c = Number  of  soldiers  who 

d = Number  of  soldiers  who 

Fail 

Pass  performance  test  and 

Fail  performance  test  and 

Fail  written  item 

Fail  written  item 

Agreement  Index  = ad  - be 


If  the  Agreement  Index  calculated  equals  any  positive  number,  then  the 
item  is  satisfactory.  If  the  Agreement  Index  calculated  equals  zero  or  is 
negative,  the  item  is  unsatisfactory  and  needs  revision 


Consider  the  following  example  of  a four  item  Scoreable  Unit: 


PERFORMANCE  TEST 


ITEM  Pass 
1 Fail 


Pass  Fall 
5 0 

0 5 


Agreement  Index  = (5) (5)  - (0) (0)  = 25 
Item  is  satisfactory. 


PERFORMANCE  TEST 


ITEM  Pass 
2 Fail 


Pass  Fall 
4 2 

1 3 


Agreement  Index  = (4) (3)  - (2)(1)  = 10 
Item  is  satisfactory. 


PERFORMANCE  TEST 


Pass 

Fail 

ITEM 

Pass 

3 

2 

3 

Fail 

2 

3 . 

Agreement  Index  = (3) (3)  - (2)  (2)  = 5 
Item  is  satisfactory. 


PERFORMANCE  TEST 


Pass  Fail 

ITEM  Pass 
4 Fail 


Agreement  Index  * (3)  (2)  - (3) (2)  = 0 
Item  is  unsatisfactory. 


The  Agreement  Indices  between  the  performance  test  and  items  1,  2,  and  3 
are  positive;  therefore  these  items  are  satisfactory.  Since  the  Agreement 
Index  for  item  4 is  zero,  it  is  unsatisfactory.  Item  4 must  be  examined 
for  possible  revision  based  on  the  same  content. 


f.  Repeat  the  procedure  presented  in  la  - le  for  the  other  Scoreable 
Unit  to  be  validated  against  a performance  test  of  the  corresponding  task. 
2.  Try  out  each  written  Scoreable  Unit  on  30  soldiers. 

a.  Obtain  samples  of  soldiers  for  Phase  II  validation  of  written 
Scoreable  Units.  The  samples  may  be  representative  soldiers,  if  they  are 
available,  or  a mixture  of  AIT  students  and  NCOs.  The  important  considera- 
tion is  to  obtain  a spread  of  ability  to  perform  the  tasks.  An  ideal 
mixture  of  the  sample  without  representative  soldiers  is  that  one-third 

of  the  soldiers  be  AIT  students  in  the  first  2-3  weeks  of  the  course,  one- 
third  of  the  soldiers  be  AIT  students  about  to  graduate  from  the  course, 
and  one-third  of  the  soldiers  be  NCOs  with  relevant  field  experience.  The 
validation  procedures  for  both  types  of  samples  of  soldiers  are  the  same. 

b.  Administer  self-ratings  of  ability  to  perform  the  tasks  covered  by 
the  Scoreable  Units.  The  self-ratings  are  used  to  identify  groups  of 
experts  and  novices.  Experts  are  those  soldiers  who  rate  themselves  high 
on  ability  to  perform  the  task  and  novices  are  those  soldiers  who  rate 
themselves  low.  A soldier  may  be  in  the  expert  group  for  one  task  and 

in  the  novice  group  for  another  task.  The  tasks  should  be  described  in 
sufficient  detail  so  that  the  soldiers  can  provide  accurate  self-ratings. 
Provide  each  soldier  with  a detailed  description  of  the  task  that  specifies 
the  task  conditions  and  standards,  and  then  asks  each  soldier  to  answer  a 
self-rating  question.  Two  prototype  formats  are  suggested. 


(l)*How  well  can  you  perform  the  task? 

0.  Not  familiar  with  the  task. 

1.  Not  very  well. 

2.  Fairly  well. 

3.  Very  well. 


(2)  To  perform  the  complete  task,  how  much  technical  guidance 
would  you  require  from  someone  who  knows  how  to  perform  the  task? 

0.  Not  familiar  with  the  task. 

1.  Lots. 

2 . Some . 

3.  None. 


For  validating  written  Scoreable  Units,  those  who  respond  0 and  1 are 
called  novices;  and  those  who  respond  2 and  3 are  called  experts. 


c._  Administer  the  Scoreable  Units  to  the  experts  and  novices  after  the 


ratings  have  been  obtained.  Identify  groups  of  experts  and  novices  on  the 
basis  of  the  self-ratings.  Exercise  caution  in  interpreting  data  for  those 
Scoreable  Units  for  which  fewer  than  fifteen  self-rated  experts  have  been 
identified.  In  these  cases  even  one  inaccurate  expert  self-rating  can  have 
a large  Influence  on  the  value  of  the  Agreement  Index.  Test  the  Scoreable 
Unit  on  additional  self-rated  experts  until  a total  of  at  least  fifteen  is 
obtained. 

d.  Compute  the  agreement  between  self-rating  (expert  or  novice)  and 
each  item  (pass  or  fall)  of  the  Scoreable  Unit.  (A  summary  form  to  be 
used  for  ITEG  submitted  is  provided  on  p.  60,  a,  b.) 
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(1)  Determine  the  Agreement  Index  between  self-rating  (expert  or 
novice)  and  each  item  (pass  or  fail)  of  the  Scoreable  Unit.  This  Agreement 
Index  indicates  how  well  an  item  reflects  mastery  on  the  task  tested  by  the 
Scoreable  Unit.  For  each  item  in  the  Scoreable  Unit  prepare  a table  as 
shown  below.  Examples  of  how  to  compute  an  Agreement  Index  between  self- 
rating  and  items  are  provided  in  paragraphs  d(3)  and  d(4).  below: 

SELF-RATING 


Expert 


Novice 


a * Number  of  soldiers  who 
rate  themselves  as 
Experts  and  pass  item 

b = Number  of  soldiers  who 
rate  themselves  as  Novices 
and  pass  item 

c = Number  of  soldiers  who 
rate  themselves  as 
Experts  and  fall  item 

d = Number  of  soldiers  who 
rate  themselves  as  Novices 
and  fail  item 

Agreement  Index  = ad  - be 


(2)  Determine  if  more  experts  pass  than  fail  each  item.  The  number 
of  experts  passing  an  item  should  be  greater  than  the  number  of  experts  fail 
ing  an  item,  that  is,  a is  greater  than  c.  If  this  relationship  is  not 
satisfied,  the  item  may  be  too  difficult  when  operationally  administered 
to  representative  soldiers. 


(3)  Consider  the  following  example  of  a four  item  Scoreable  Unit. 


SCORES  Ox-  EXPERTS  AND  NOVICES  ON  A FOUR  ITEM  SCOREABLE  UNIT 


ITEMS 


EXPERTS 

1 

2 

3 

4 

Total 

Soldier  1 

1 

1 

1 

1 

4 

Soldier  2 

1 

1 

1 

1 

4 

Soldier  3 

1 

0 

1 

1 

3 

Soldier  4 

1 

1 

1 

1 

4 

Soldier  5 

0 

1 

1 

1 

3 

Soldier  6 

0 

1 

0 

1 

2 

Soldier  7 

0 

0 

1 

1 

2 

Soldier  8 

0 

1 

1 

1 

3 

Soldier  9 

0 

1 

0 

1 

2 

Soldier  10 

0 

0 

1 

1 

2 

Soldier  11 

1 

1 

1 

0 

3 

Soldier  12 

1 

1 

1 

1 

4 

Soldier  13 

0 

1 

1 

1 

3 

Number  Passing 

6 

10 

11 

12 

39  Checks 

Number  Failing 

7 

3 

2 

1 

NOVICES 

1 

3 

4 

Total 

Soldier  14 

1 



0 

0 

1 

2 

Soldier  15 

1 

0 

1 

1 

3 

Soldier  16 

1 

0 

1 

1 

3 

Soldier  17 

1 

0 

0 

1 

2 

Soldier  18 

1 

0 

0 

1 

2 

Soldier  19 

0 

0 

0 

1 

1 

Soldier  20 

1 

0 

0 

1 

2 

Soldier  21 

0 

0 

1 

0 

1 

Soldier  22 

0 

0 

0 

1 

1 

Soldier  23 

0 

0 

0 

0 

0 

Soldier  24 

1 

0 

1 

1 

3 

Soldier  25 

1 

0 

0 

1 

2 

Soldier  26 

1 

1 

0 

0 

2 

Soldier  27 

1 

0 

0 

0 

1 

Soldier  28 

0 

0 

0 

0 

0 

Soldier  29 

0 

1 

1 

0 

2 

Soldier  30 

0 

0 

0 

0 

0 

Number  Passing 

10 

2 

5 

10 

27  Checks 

Number  Failing 

7 

15 

12 

7 

(a)  Sort  experts  from  novices  on  the  basis  of  their  self- 
ratings;  record  ea  h soldier's  performance  on  each  item;  record  a "1"  for 
each  item  passed  and  a "0"  for  each  item  failed. 


(b)  Count  the  total  number  of  items  each  expert  passed. 
(These  scores  are  shown  in  the  last  column  labelled  "Total.") 


(c)  Count  the  number  of  experts  who  passed  and  failed  each  item. 
(These  numbers  are  recorded  in  the  rows  labelled  "Number  Passing"  and 
"Number  Failing.")  The  sum  of  the  Number  Passing  and  the  Number  Failing 
for  each  item  must  equal  the  number  of  experts.  (If  an  item  in  a Scoreable 
Unit  is  omitted,  count  that  item  as  failed.) 


(d)  Check  the  accuracy  of  the  counts  by  summing  the  Total  column 
and  summing  the  Number  Passing  row;  these  numbers  must  be  the  same. 


(e)  Repeat  steps  (b) , (c) , and  (d)  for 
Passing  plus  Number  Failing  in  step  (c)  must  equal 


novices;  the  Number 
the  number  of  novices. 


(f)  Compare  self-rating  and  item  performance  as  shown  in  the 
following  two  by  two  tables.  Soldiers'  performance  (pass  or  fail)  on  items 
should  agree  with  their  self-rating  on  the  task.  This  will  be  indicated  by 
a positive  Agreement  Index. 
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SELF-RATING 


AGREEMENT  INDEX 


a = 6 

b = 10 

c = 7 

d = 7 

6(7)  - 10(7)  = -28 


10 

2 

3 

15 

10(15)  - 2(3)  = 144 


11 

5 

2 

12 

11(12)  - 5(2)  = 122 


12 

10 

1 

7 

12(7)  - 10(1)  = 74 


The  Agreement  Index  for  item  1 is  less  than  zero;  therefore,  the  self- 
rating Agreement  Index  is  unsatisfactory  for  item  1.  The  self-rating 
Agreement  Index  for  items  2,  3,  and  4 is  positive;  these  three  items  are 
satisfactory  with  respect  to  the  requirement  that  items  agree  with  self- 
rating. 

(g)  Compare  the  number  of  experts  passing  and  failing  each  item. 
More  experts  must  pass  than  fail  each  item  (a  > c) . Since  fewer  experts 
passed  than  failed  item  1,  it  is  unsatisfactory  with  respect  to  this  require- 
ment. Since  more  experts  passed  than  failed  items  2,  3,  and  4;  these  three 
items  arc  satisfactory  according  to  this  requirement. 


(4)  Consider  the  following  example  of  a five  item  Scoreable  Unit. 


SCORES  Of  EXPERTS  AND  NOVICES  ON  A FIVE  ITEM  SCOREABLE  UNIT 


! 


I-  j 

n 


ITEMS 


EXPERTS 

1 

2 

3 

4 

5 

Total 

Soldier  1 

1 

1 

1 

1 

1 

5 

Soldier  2 

1 

1 

1 

1 

1 

5 

Soldier  3 

1 

1 

1 

1 

0 

4 

Soldier  4 

1 

1 

1 

0 

0 

3 

Soldier  5 

1 

0 

1 

1 

0 

3 

Soldier  6 

1 

1 

0 

1 

0 

3 

Soldier  7 

1 

1 

1 

1 

0 

4 

Soldier  8 

1 

1 

1 

0 

0 

3 

Soldier  9 

1 

1 

1 

1 

0 

4 

Soldier  10 

1 

1 

1 

1 

0 

4 

Soldier  11 

0 

1 

0 

1 

1 

3 

Soldier  12 

0 

1 

0 

1 

1 

3 

Soldier  13 

0 

1 

0 

0 

1 

2 

Soldier  14 

0 

0 

0 

0 

1 

1 

Soldier  15 

0 

0 

0 

1 

1 

2 

Soldier  16 

1 

0 

0 

0 

0 

1 

Number  Passing 

11 

12 

9 

11 

7 

50 

Number  Failing 

5 

4 

7 

ITEMS 

5 

9 

NOVICES 

1 

2 

3 

4 

5 

Total 

Soldier  17 

0 

1 

1 

1 

0 

3 

Soldier  18 

1 

0 

0 

0 

1 

2 

Soldier  19 

0 

0 

0 

0 

0 

0 

Soldier  20 

0 

1 

0 

1 

0 

2 

Soldier  21 

0 

0 

1 

0 

1 

2 

Soldier  22 

0 

0 

1 

0 

0 

1 

Soldier  23 

0 

0 

1 

1 

0 

2 

Soldier  24 

0 

1 

0 

0 

1 

2 

Soldier  25 

1 

0 

0 

1 

1 

3 

Soldier  26 

0 

0 

1 

0 

1 

2 

Soldier  27 

1 

0 

0 

1 

1 

3 

Soldier  28 

1 

1 

1 

0 

1 

4 

Soldier  29 

0 

1 

1 

1 

1 

4 

Soldier  30 

0 

1 

1 

0 

1 

3 

Number  Passing 

4 

6 

8 

6 

9 

33 

Number  Failing 

10 

8 

6 

8 

5 

( 
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(a)  Sort  experts  from  novices  on  the  basis  of  their  self- 
ratings;  record  each  soldier's  performance  on  each  item;  record  a "1"  for 
each  item  passed  and  a "0"  for  each  item  failed. 


(b)  Count  the  total  number  of  items  each  expert  passed.  (These 
scores  are  shown  in  the  last  column  labelled  "Total.") 


(c)  Count  the  number  of  experts  who  passed  and  failed  each  item. 
(These  numbers  are  recorded  in  the  rows  labelled  "Number  Passing"  and 
"Number  Failing.")  The  sura  of  the  Number  Passing  and  the  Number  Failing 
for  each  item  must  equal  the  number  of  experts.  (If  an  item  in  a Score- 
able  Unit  is  omitted,  count  that  item  as  failed.) 


(d)  Check  the  accuracy  of  the  counts  by  summing  the  Total  Column 
and  summing  the  Number  Passing  row;  these  numbers  must  be  the  same. 


(e)  Repeat  steps  (b) , (c) , and  (d)  for  novices;  the  Number 
Passing  plus  Number  Failing  in  step  (c)  must  equal  the  number  of  novices. 


(f)  Compare  self-rating  and  item  performance  as  shown  in  the 
following  two  by  two  tables.  Soldiers'  performance  (pass  or  fail)  on  items 
should  agree  with  their  self-ratings  on  the  task.  This  will  be  indicated  by 
a positive  Agreement  Index. 
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SELF-RATING 


AGREEMENT  INDEX 
ad  - be  > 0 


1 

i 

Expert 

Novice 

j ITEM 

Pass  1 

a = 11 

b = 4 

• 

1 1 

Fail  I 

c = 5 

d = 10 

11(10)  - 4(5)  = 90 

! 

1 

1 

i 

i 

1 

Expert 

Novice 

i ITEM 

Pass 

12 

6 

i 2 

Fail 

t 4 

8 

12(8)  - 6(4)  = 72 

Expert 

Novice 

! ITEM 

Pass 

9 

8 

3 

Fail 

7 

6 

9(6)  - 8(7)  = - 2 

Expert 

Novice 

ITFJi 

Pass 

11 

6 

4 

Fail 

5 

8 

11(8)  - 6(5)  = 58 

Expert 

Novice 

ITEM 

Pass 

7 

9 '' 

5 

Fall 

9 

5 

7(5)  - 9(9)  = -46 

The  Agreement  Index  for  items  3 and  5 is  less  than  zero;  while  that 
for  items  1,  2,  and  4 is  positive.  Therefore,  items  3 and  5 are  un- 
satisfactory with  respect  to  this  requirement. 


(g)  Compare  the  number  of  experts  passing  and  failing  each  item. 
More  experts  passed  than  failed  items  1,  2,  3,  and  4;  however,  more 
experts  failed  than  passed  item  5.  Therefore,  item  5 is  unsatisfactory 
with  respect  to  this  requirement. 
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e.  Compute  Che  Item-Scoreable  Unit  agreement  for  each  Scoreable  Unit. 

If  a Scoreable  Unit  In  Che  Written  Component  consists  of  four  or  fewer 
items,  then  obtain  the  extent  of  agreement  between  all  pairs  of  items  in 
the  Scoreable  Unit.  If  a Scoreable  Unit  in  the  Written  Component  consists 
of  five  or  more  items,  Chen  obtain  Che  extent  of  agreement  between  each 
item  and  the  total  score  on  the  Scoreable  Unit.  CA  form  to  be  used  for  ITEG 
submittal  is  provided  on  p, 61, a, b.  Additional  forms  that  the  TDA  may  find 
useful  in  performing  the  item  analysis  are  provided  on  pp.  64-65.) 

(1)  Scoreable  Units  with  four  or  fewer  items. 

(a)  Compute  the  Agreement  Indices  between  all  possible  pairs 
of  items  from  a Scoreable  Unit.  The  following  table  is  a general 
illustration  of  a two  by  two  table: 


ITEM  1 


Pass 

Fail 

a = Number  who 

b = Number  who 

Pass 

Pass  Item  1 

Fail  Item  1 

and 

and 

Pass  Item  2 

Pass  Item  2 

c - Number  who 

d = Number  ^^^ho 

Fail 

Pass  Item  1 

Fail  Item  1 

and 

and 

Fail  Item  2 

Fail  Item  2 
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(b)  Consider  the  four  item  Scoreable  Unit  presented  in 


paragraph  d.(3)  (p, 22-25  );  the  following  table  lists  the  performance  of 
all  thirty  soldiers.  The  expert-notice  distinction  has  been  deleted  because 
it  is  not  used  in  this  evaluation  of  the  Scoreable  Units.  The  following 
is  a description  of  how  to  evaluate  the  four  item  Scoreable  Unit. 


5 ‘ 


i: 


ITEMS 


SOLDIERS 

] 

2 

3 

4 

Total 

Soldier  1 

1 

1 

1 

1 

4 

Soldier  2 

1 

1 

1 

1 

4 

Soldier  3 

i 

0 

1 

1 

3 

Soldier  A 

1 

i. 

1 

1 

4 

Soldier  5 

0 

1 

1 

1 

3 

Soldier  6 

0 

1 

0 

1 

2 

Soldier  7 

0 

0 

1 

1 

2 

Soldier  8 

0 

1 

1 

1 

3 

; ■ 

Soldier  9 

u 

1 

0 

1 

2 

Soldier  10 

0 

0 

1 

1 

2 

Soldier  11 

J 

1 

1 

0 

3 

r 

i 

Soldier  12 

1 

1 

1 

1 

4 

f 

i . 

Soldier  13 

0 

1 

1 

1 

3 

1 

Soldier  14 

1 

0 

0 

1 

2 

F * 
h i 

Soldier  15 

1 

0 

1 

1 

3 

V 

Soldier  16 

1 

0 

1 

1 

3 

Soldier  17 

1 

0 

0 

1 

2 

il 

Soldier  18 

1 

0 

0 

1 

2 

Soldier  19 

0 

0 

0 

1 

1 

’<  i 

Soldier  20 

1 

0 

0 

1 

2 

Soldier  21 

0 

0 

1 

0 

1 

Soldier  22 

0 

u 

0 

1 

1 

!• 

Soldier  23 

0 

0 

0 

0 

0 

Soldier  24 

1 

0 

1 

1 

3 

. 1 
: ( 

Soldier  25 

1 

0 

0 

1 

2 

Soldier  26 

1 

1 

0 

0 

2 

1 1 

Soldier  27 

1 

0 

0 

0 

1 

1 

Soldier  28 

0 

0 

0 

0 

0 

! 

Soldier  29 

0 

1 

1 

0 

2 

Soldier  30 

0 

0 

0 

0 

0 

■ 

‘i 

Number  Passing 

16 

12 

16 

22 

Number  Failing 

14 

18 

14 

8 

E; 


fc)  Prepare  a two  by  two  table  for  each  pair  of  Items. 
Consider  Item  1 and  Item  2.  Count  the  total  number  of  soldiers  who: 


(a)  passed  both  Items  1 and  2,  Cb)  passed  Item  1 and  failed  Item  2,  (c) 
failed  Item  1 and  passed  Item  2,  and  (d)  failed  both  Items  1 and  2.  For 
example  In  the  above  table,  solder  1 passed  both  Item  1 and  Item  2;  soldier  3 
passed  Item  1 and  failed  Item  2;  soldier  5 failed  Item  1 and  passed  Item  2; 
and  soldier  7 failed  both  Item  1 and  Item  2.  Six  soldiers  passed  Item  1 
and  passed  Item  2.  Ten  soldiers  passed  Item  1 and  frl]ed  Item  2.  Six 
soldiers  failed  item  1 and  passed  Item  2.  Eight  soldiers  failed  Item  1 
and  failed  item  2.  This  may  be  expressed  in  a two  by  two  table  as  follows: 


item  2 AGREEMENT  INDEX 

ad  - be  > 0 


A similar  procedure  is  followed  lor  the  other  item  pair  comparisons 
In  the  case  of  items  1 and  3,  count  the  total  number  of  soldiers  who 
passed  both  item  1 and  item  3,  passed  item  1 and  failed  item  3,  failed 
item  1 and  passed  item  3,  and  failed  both  item  1 and  item  3.  Nine 
soldiers  passed  item  1 and  passed  item  3,  7 passed  item  1 and  failed 
item  3,  7 failed  item  1 and  passed  item  3,  and  7 failed  both  item  1 and 
item  3.  This  may  be  expressed  in  a two  by  two  table  as  follows: 


ITEM  3 


AGREEMENT  INDEX 


Pass 

Fail 

Total 

Pass 

9 

7 

16 

Fail 

7 

7 

14 

Total 

16 

14 

30 

9(7)  - 7(7)  = 14 


ITEM  4 


Pass 

Fail 

Total 

ITEM 

Pass 

13 

3 

16 

1 

Fail 

9 

5 

14 

Total 

22 

8 1 

30 

13(5)  - 3(9)  = 38 


ITEM  3 


Pass 

Fail 

Total 

Pass 

9 

3 

12 

Fail 

7 

11 

18 

Total 

16 

14 

30 

9(11)  - 3(7)  = 78 


ITEM  4 


Pass 

Fail 

Total 

Pass 

9 

3 

12 

Fail 

13 

5 

18 

Total 

22 

8 

30 

9(5)  - 3(13)  = 6 


ITEM  4 


Pass  Fail  Total 


3 


9 I 5 
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C2)  Scoreable  Dolts  with  five  or  more  Items. 

Ca)  Compare  Items  to  the  total  score  on  the  Scoreable  Chit. 
Consider  the  five  item  Scoreable  Unit  presented  in  paragraph  d(4)  (p.  25-28), 
The  following  is  a description  of  how  to  evaluate^  the  Scoreable  Unit. 

(b)  Prepare  a table  which  indicates  the  number  of  soldiers 
(experts  and  novices  combined)  who  passed  0 items,  1 item,  2 items,  3 items, 
4 items  and  5 items.  (If  there  were  more  items  in  the  Scoreable  Unit, 
then  the  table  would  have  more  columns.) 


Items  passed 0_ 

Number  of  Soldiers  1 


_2 3_ 

8 10 


(c)  Determine  high  and  low  scoring  groups  by  dividing  the 
thirty  soldiers  into  two  groups  that  are  as  close  as  possible  to  the  same  size 
In  this  example,  the  high  scoring  group  contains  18  soldiers  who  passed 
3,  4,  or  5 items.  The  low  scoring  group  contains  12  soldiers  who  passed 
0,  1,  or  2 items.  This  split  resulted  in  two  groups  that  were  as  close  as 

t 

possible  to  the  same  size. 


The  following  table  presents  the  thirty  soldiers  by  high  and  low 
scoring  groups. 


1 


I '■ 


ITEMS 


HIGH 

1 

2 

3 

4 

5 

Total 

Soldier  1 

1 

1 

1 

1 

1 

5 

Soldier  2 

1 

1 

1 

1 

1 

5 

Soldier  3 

1 

1 

1 

1 

0 

4 

Soldier  4 

1 

1 

1 

0 

0 

3 

Soldier  5 

1 

0 

1 

1 

0 

3 

Soldier  6 

1 

1 

0 

1 

0 

3 

Soldier  7 

1 

1 

1 

1 

0 

4 

Soldier  8 

1” 

1 

1 

0 

0 

3 

Soldier  9 

1 

1 

1 

1 

0 

4 

Soldier  10 

1 

1 

1 

1 

0 

4 

Soldier  11 

0 

1 

0 

1 

1 

3 

Soldier  12 

0 

1 

0 

1 

1 

3 

Soldier  17 

0 

1 

1 

1 

0 

3 

Soldier  25 

1 

0 

0 

1 

1 

3 

Soldier  27 

1 

0 

0 

1 

1 

3 

Soldier  28 

1 

1 

1 

0 

1 

4 

Soldier  29 

0 

1 

1 

1 

1 

4 

Soldier  30 

0 

1 

1 

0 

1 

3 

Number  passing 

13 

15 

13 

14 

9 

64 

Number  failing 

5 

3 

5 

4 

9 

ITEMS 

LOW 

1 

2 

3 

4 

5 

Total 

Soldier  13 

0 

1 

0 

0 

1 

2 

Soldier  14 

0 

0 

0 

0 

1 

1 

Soldier  15 

0 

0 

0 

1 

1 

2 

Soldier  16 

1 

0 

0 

0 

0 

1 

Soldier  18 

1 

0 

0 

0 

1 

2 

Soldier  19 

0 

0 

0 

0 

0 

0 

Soldier  20 

0 

1 

0 

1 

0 

2 

Soldier  21 

0 

0 

1 

0 

1 

2 

Soldier  22 

0 

0 

1 

0 

0 

1 

Soldier  23 

0 

0 

1 

1 

0 

2 

Soldier  24 

0 

1 

0 

0 

1 

2 

Soldier  26 

0 

0 

1 

0 

1 

2 

Number  passing 

2 

3 

4 

3 

7 

19 

Number  falling 

10 

9 

8 

9 

5 
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64  checks 


19  checks 


(^'  Determine  the  Agreement  Index  between  scoring  group  (high  or  low) 
and  each  item  (pass  or  fail)  of  the  Scoreable  Unit*  Count  for  each  Item 
the  number  of  high  and  low  scorers  who  passed  and  failed  the  item. 

(These  numbers  are  already  indicated  in  the  table.)  The  following 
tables  are  used  to  obtain  the  Agreement  Indices. 


SCORING  GROUP 


AGREEMENT  INDEX 


ITEM 

1 


ITEM 

2 


ITEM 

3 


ITEM 

4 


13(10)  - 2(5)  - 120 


15(9)  - 3(3)  = 126 


13(8)  - 4(5)  - 84 


14(9)  - 3(4)  = 114 


item  Pass 
5 Fail 


High  Low 
9 7 

9 5 
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9(5)  - 7(9)  - -18 


(e)  Evaluate  Items  In  Scoreable  Unit  with  5 or  more  items  by 
computing  the  Item-Scoreable  Unit  agreement.  In  the  above  example  of  a 
5 item  Scoreable  Unit,  the  Agreement  Indices  for  items  1,  2,  3 and  4 are 
all  greater  than  zero,  indicating  that  these  items  are  satisfactory  with 
respect  to  item-Scoreable  Unit  agreement.  The  Agreement  Index  for  item 
5 is  less  than  zero;  therefore,  item  3 is  unsatisfactory  with  respect  to 
Item-Scoreable  Unit  agreement. 

f.  Evaluate  the  items  within  each  Scoreable  Unit.  (A  summary  form 
to  be  used  for  ITEG  submittal  is  located  on  p.  62,  a,  b) 

(1)  Identify  those  items  found  to  be  satisfactory  and  unsatis- 
factory with  respect  to  self-rating  (as  determined  in  paragraph  d). 

(2)  Identify  those  items  found  to  be  satisfactory  and  unsatis- 
factory with  respect  to  item-Scoreable  Unit  agreement  (as  determined  in 
paragraph  e) . 

(3)  Determine  those  items  which  are  acceptable  and  unacceptable 
by  combining  the  information  obtained  in  (1)  and  (2);  if  an  item  is 
satisfactory  in  (1)  and  (2),  it  is  acceptable  in  its  present  form.  If 
an  item  is  unsatisfactory  in  both  (1)  and  (2) , it  is  unacceptable  in  its 
present  form  and  must  be  revised.  If  an  item  is  unsatisfactory  in 
either  (1)  or  (2),  it  must  be  reviewed  to  determine  whether  revision  is 
required. 


3 


ITEM-SCOREABLE 
UNIT  AGREEMENT 

Satisfactory 

Unsatisfactory 


SELF-RATING 

Satisfactory  Unsatisfactory 


ACCEPT 

REVIEW 

REVIEW 

REVISE 

ACCEPT  an  item  means  retain  it  in  its  present  form.  REVISE  means  change 
the  item  to  correct  deficiencies.  REVIEW  means  consider  all  available 
information  and  then  decide  if  and  how  to  revise. 


(a)  Consider  the  example  of  the  four  item  Scoreable  Unit 
introduced  previously.  Item  1 was  unsatisfactory  with  respect  to  both 
aspects  of  self-rating  and  item-Scoreable  Unit  agreement.  Item  2 was 
satisfactory  with  respect  to  self-rating  but  not  item-Scoreable  Unit 
agreement.  Items  3 and  4 were  satisfactory  with  respect  to  both  self- 
rating and  item-Scoreable  Unit  agreement.  This  information  is  summar- 
ized in  the  table  below. 


1 

i 

I 


i 


ITEM-SCOREABLE 

SELF- 

-RATING 

UNIT  AGREEMENT 

Satisfactory 

Unsatisfactory 

Satisfactory 

ACCEPT 

REVIEW 

items  3,  4 

no  items 

Unsatisfactory 

REVIEW 

REVISE 

item  2 

item  1 

Therefore,  items  3 and  4 are  acceptable  in  their  present  form.  Item  1 
is  unacceptable  in  its  present  form  and  must  be  revised.  Item  2 must  be 
reviewed.  A review  of  item  2 would  probably  suggest  retaining  item  2 in 
its  present  form.  Item  2 was  unsatisfactory  with  respect  to  item- 
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Scoreable  Unit  agreement  because  of  Its  negative  Agreement  Index  vhen 
paired  with  Item  1.  However,  Item  1 was  found  to  be  unacceptable  with 
respect  to  both  Item-Scoreable  Unit  and  Item-self  rating.  Therefore, 
the  negative  Agreement  Index  between  Items  1 and  2,  can  probably  be 
attributed  to  deficiencies  In  Item  1. 

(b)  Consider  the  example  of  the  five  Item  Scoreable  Unit. 
Item  5 was  unsatisfactory  with  respect  to  both  aspects  of  self-rating 
and  item-Scoreable  Unit  agreement.  Item  3 was  unsatisfactory  with 
respect  to  one  aspect  of  self-rating  but  not  item-Scoreable  Unit  agree- 
ment. Items  1,  2,  and  4 were  satisfactory  with  respect  to  both  self- 
rating and  item-Scoreable  Unit  agreement.  This  Information  is  summar- 
ized in  the  table  below. 


ITEM-SCOREABLE 

SELF- 

•RATING 

UNIT  AGREEMENT 

Satisfactory 

Unsatisfactory 

Satisfactory 

ACCEPT 

REVIEW 

Items  1,2,4 

item  3 

Unsatisfactory 

REVIEW 

REVISE 

No  Items 

item  5 

Therefore,  items  1,  2 and  4 are  acceptable  in  their  present  form.  Item 
5 Is  unacceptable  In  Its  present  form  and  must  be  revised.  Item  3 must 
be  reviewed.  Item  3 was  found  to  be  unsatisfactory  with  respect  to 
self-rating.  Unless  a reasonable  explanation  exists  for  this  finding. 
Item  3 must  be  reviewed  for  possible  revision. 
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g.  Determine  ac-eptablllty  to  expert  soldiers.  For  each  Scoreable 

^ Unit  at  least  80  percent  of  the  soldiers  identified  as  experts  on  the 

j task  from  which  the  Scoreable  Unit  was  derived,  must  accept  the  Score- 

I 

! able  Unit  as  a fair  test  of  their  ability  to  perform  on  the  job  by 

; answering  "yes"  on  question  sheet  5 (p.  48)*  Any  Scoreable  Unit  accept- 

ed by  less  than  80  percent  of  the  experts  must  be  revised.  (A  summary  form 
for  ITEG  submittal  is  provided  at  p.  59,  a,  b.). 

h.  Identify  all  items  and  Scoreable  Units  requiring  revision. 

These  Include  those  items  in  Scoreable  Units  identified  in  f.  during  the 
evaluation  process.  If  a Scoreable  Unit  was  found  unacceptable  in  g.  to 

experts  and  has  already  had  items  identified  as  unacceptable  in  f.,  then 

correction  of  these  items  in  the  Scoreable  Unit  is  sufficient.  However, 
if  a Scoreable  Unit  was  found  unacceptable  in  g.,  but  had  no  items 
Identified  as  unacceptable  in  f.,  then  the  entire  Scoreable  Unit  must  be 
re-examined  to  Identify  the  source  of  the  problem.  Recall  that  any 
revision  in  an  item  cannot  change  the  content  covered  in  the  original 
item. 

n 

1 

I 3 . Revise  Scoreable  Units  with  unacceptable  items . 

i 

i 

a.  Develop  a new  form  for  each  item  found  to  be  unacceptable. 

> 

? ; 

I I 

• b.  Test  revised  items  on  5 experts.  Have  the  experts  inspect  both  j 

j versions  of  the  item  (the  unacceptable  original  version  and  the  revised  | 

version) . Have  experts  decide  if  both  items  have  the  same  content  and 
j which  version  is  better.  Do  not  identify  either  version  as  the  original 

I 
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or  revised.  The  experts  also  key  each  of  the  two  versions  of  the  item. 


c.  Require  that  at  least  80  percent  of  the  experts  agree  that  the 
content  Is  not  changed.  Require  also  that  at  least  80  percent  of  the 
experts  agree  on  the  scoring  key  of  the  new  version  of  the  item. 


d.  Identify  all  Sgoreable  Units  where  one-third  or  less  of  the 
items  required  revision.  For  these  Scoreable  Units  no  additional  tryout 
is  required;  i.e.,  the  expert  evaluation  conducted  in  b.  is  sufficient. 


e.  Try  out  the  Scoreable  Units  where  more  than  one-third  of  the 
items  required  revision  on  at  least  20  soldiers.  For  these  Scoreable 
Units  such  a large  part  of  the  Scoreable  Unit  was  unacceptable  that  the 
expert  tryout  alone  is  not  sufficient.  The  new  version  of  the  Scoreable 
Unit  must  be  administered  to  at  least  20  soldiers  and  the  procedures 
described  in  2a.  - 2h.  must  be  repeated;  that  is,  obtain  self-ratings, 
compute  item-self  rating  and  item-Scoreable  Unit  agreement,  evaluate 
items,  determine  acceptability,  and  revise  as  required. 


4.  Try  out  any  Written  Component  containing  unique  instructions  with 
representative  TCP  and  soldiers.  If  a Written  Component  contains  group 
paced  Scoreable  Units,  considerable  use  of  technical  manuals,  simula- 
tors, or  any  type  of  materials  or  Instructions  that  could  result  in 
feasibility  problems,  the  TDA  must  try  out  the  entire  component  on  as 
large  a group  of  representative  soldiers  as  may  be  encountered  during 
operational  testing.  For  example,  if  slides  are  used  as  part  of  a 
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Scoreable  Unit  and  c etailed  viewing  is  required,  then  the  maximum  view- 
ing distance  must  be  demonstrated  prior  to  operational  testing. 


a.  Try  out  entire  Written  Component  using  operational  answer  sheets 


and  all  necessary  administrative  instructions. 


b.  Determine  administrative  feasibility. 


that  do  not  provide  standardized  testing  conditions  for  all  soldiers. 


e.  Adjust  the  administrative  instructions  for  operational  adminis- 


trati'SA  or  revise  Scoreable  Unit  to  correct  for  potential  administrative 
problems.  Any  significant  change  in  the  Scoreable  Unit  requires  confirma 
tion  by  experts  that  the  content  has  not  been  altered.  If  the  Scoreable 
Unit  is  revised,  follow  the  steps  indicated  in  3a.  - 3e. 


. Assemble  written  component  to  include  a summary  of  tryout  results. 


6 . Submit  written  component  to  ITEG;  revise  as  required . 


7.  Assemble  final  written  component. 


NAME 


f 


QUESTION  SHEET  1 

IDENTIFICATION  OF  ESSENTIAL  AND  KEY  BEHAVIORS 


Yes  No 


1.  Is  the  task  covered  by  the  Scoreable  Unit  a critical 
job  requirement  at  this  skill  level? 

Based  on  your  experience  in  performing  the  task,  and  on 
your  experience  in  watching  other  people  perform  the  task, 
please  answer  questions  2 and  3,  below. 

2.  Identify  Essential  Behaviors 

Does  the  task  analysis  list  all  the  Essential  Behav- 
iors (decisions  or  actions)  necessary  for  task  per- 
formance? 

If  not,  please  specify  which  Essential  Behaviors 
(decisions  or  actions)  should  be: 

a.  Added 

Which  one(s)? 

b.  Deleted 
Which  one(s)? 

c . Revised 

Which  one(s),  and  how? 

3.  Identify  Key  Behaviors 

Does  the  task  analysis  identify  all  the  Key  Behaviors 
(which  have  serious  consequences  to  personnel  or 
equipment,  or  are  frequent  sources  of  failure)? 

If  not,  please  specify  which  Key  Behaviors  should  be: 

a.  Added 

Which  one(s)? 

b.  Deleted 
Which  one(s)? 

c.  Revised 

Which  one(s),  and  how? 


4 
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NAME 


QUESTION  SHEET  2 

EXPERT  AND  RATER  QUESTIONS  FOR  HANDS-ON  SCOREABLE  UNITS 

Please  answer  the  following  questions  for  each  Scoreable  Unit  Immediately 
after  it  is  administered. 

Yes  No 

1.  Is  the  task  covered  by  the  Scoreable  Unit  a critical 

job  requirement  at  the  skill  level?  

If  not,  why  not? 

2.  Does  the  Scoreable  Unit  provide  a fair  measure  of 

job  requirements?  

If  not,  what  changes  are  required? 

a.  Add  Performance  Measures? 

Which  ones? 

b.  Delete  Performance  Measures? 

Which  ones? 

c.  Modify  Performance  Measures? 

How? 

d.  Alter  standards? 

How? 

e.  Clarify  instructions? 

How? 

f.  Change  test  site  or  equipment  requirements? 

How? 

3.  Is  the  Scoreable  Unit  administratively  feasible? 


If  not,  what  changes  are  required? 
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NAME  

QUESTION  SHEET  3 

EVALUATION  OF  WRITTEN  SCOREABLE  UNITS 

Each  Item  should  be  based  on  a behavior  (decision  or  action)  essential  to 
performing  the  task. 

Yes  No 


1.  Does  passing  each  item  correspond  to  an  Essential 
Behavior  (decision  or  action)  necessary  for  task 
performance? 

If  not,  please  specify  which  items  should  be: 

a.  Added 

Which  one(s)? 

b.  Revised 

Which  one(s) , and  how? 

2.  Do  the  items  for  each  Key  Behavior  fully  measure 
that  point?  (Key  Behaviors  have  special  conse- 
quences to  personnel  or  equipment  or  are  frequent 
sources  of  failure) 

If  not,  please  specify  which  items  should  be: 

a.  Added 

Which  one(s)? 

b . Revised 

Which  one(s) , and  how? 

3.  Based  on  this  Scoreable  Unit,  would  you  be  confi- 
dent that  soldiers  who  answer  the  items  correctly 
are  able  to  perform  the  task? 
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NAME 


QUSTION  SHEET  5 


EXAMINEE  QUESTION  FOR  WRITTEN  SCOREABLE  UNITS 


Do  you  accept  this  set  of  items  as  a fair  measure  of  your  ability  to  perform 
the  task  as  it  is  described  in  the  Soldier's  Manual? 


Yes 


No 


Set  1 


Instructions: 


Phase  I Hands-On  Component 

Summary  of  Acceptability  to  Experts 
(Question  Sheet  2) 


(1)  Enter  the  number  of  experts  on  which  the  results 
are  based  in  the  parentheses  next  to  each  Scoreable 
Unit  number. 

I 

(2)  Enter  for  each  Scoreable  Unit  the  percent  of  experts 
responding  YES  to  each  question. 

(3)  Circle  any  entry  less  than  80  percent. 


Scoreable 


Questions 

I 2 I 


Notes 


Phase  I Hands-On  Component 

Sununary  of  Acceptability  to  Raters 
(Question  Sheet  2) 

Instructions:  (1)  Enter  the  percent  of  raters  responddlpg  YES  to 

questions  1,  2,  and  3 for  each  Scoreable  Unit. 

(2)  Circle  any  entry  less  than  75  percent. 


Scoreable 


Questions 


Phase  I Hands-On  Component 
Summary  of  Test  Results  on  Experts 


I H I.PJiPlJlJWW 


I 

I 

f 

i 


Instructions: 


(1) 


Enter  the  number  of  experts  on  which  the  results  are 
based  in  the  parentheses  next  to  each  Scoreable  Unit 
number . 


(2) 

(3) 

(4) 


Enter  for  each  Scoreable  Unit  the  percent  of  experts  GO  I 
on  the  Scoreable  Unit.  I 

Enter  for  each  Scoreable  Unit  the  percent  of  experts 
passing  each  Performance  Measure. 

Circle  any  entry  less  than  80  percent.  f 


Score- 

able 

Unit 

% 

GO 

1 

2 

3 

4 

5 

6 

% Pa 
7 

ssin 

8 

g Pe 
9 

rfon 

10 

nanc 

11 

e Me 
12 

asur 

13 

e 

14 

15 

16 

17 

18 

19 

i 

20  1 

i 

1 ( ) 

1 

2 ( ) 

1 

3 LJ_ 

1 

4 ( ) 

1 

5 ( ) 

6 ( ) 

7 ( ) 

8 ( ) 

9 ( ) 

• 

10  ( ) 

11  ( ) 

12  ( ) 

13  ( ) 

14  ( ) 

15  ( ) 

■; 

1 

16  ( ) 

« 

-J 
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Instructions: 


Phase  I Hands-On  Component 
Summary  of  Tryout  on  Novices 


(1)  Enter  the  number  of  novices  on  which  the  results 
are  based  on  the  parentheses  next  to  each  Scoreable 
Unit  number. 

(2)  Enter  the  rater  agreement  for  each  Performance 
Measure  In  each  Scoreable  Unit. 

(3)  Circle  any  entry  less  than  80  percent. 


Phase  1 Written  Component 

Summary  of  Acceptability  to  Experts 
(Question  Sheet  3) 


Instructions:  (1) 

(2) 

(3) 


Enter  the  number  of  experts  on  which  the  results  are 
based  In  the  parentheses  next  to  each  Scoreable  Unit 
number . 

Enter  for  each  Scoreable  Unit  the  percent  of  experts 
responding  YES  to  each  question. 

Circle  any  entry  less  than  80  percent. 


Scoreable 

Unit 

Q< 

1 

iestloi 

2 

IS 

3 

Notes 

1 

( 

) 

2 

( 

) 

3 

( 

) 

4 

( 

) 

5 

( 

) 

6 

( 

) 

7 

( 

) 

■ 

8 

( 

) 

B 

9 

( 

) 

10 

( 

) 

11 

J 



Phase  I Written  Component 
Summary  of  Tryout  on  Experts 


Instructions:  (1)  Enter  the  number  of  experts  on  which  the  results  are 

based  in  the  parentheses  next  to  each  Scoreable  Unit 
number . 

(2)  Enter  for  each  Scoreable  Unit  the  percent  of  experts 
passing  each  item. 

(3)  Circle  any  entry  less  than  80  percent. 


Scoreable 

Item 

Unit 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

42  ( 

) 

43  ( 

) 

44  ( 

) 

46  ( 

) 

46  ( 

) 

47  ( 

) 

48  ( 

) 

49  ( 

IQ( 

_L_ 
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Check  whether  results  apply  to: 

Local  tryout 

Field  tryout 
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Phase  II  Hands-On  Component 

Summary  of  Acceptability  to  Representative  Soldiers 
(Question  Sheet  4) 


Instructions: 


(1)  Enter  the  number  of  soldiers  on  which  the  results  are 
based  in  the  parentheses  next  to  each  Scoreable  Unit 
number . 


(2)  Enter  the  percent  soldiers  accepting  the  test  as  a 
fair  measure  of  ability. 

(3)  Circle  any  entry  less  than  65  percent. 


Scoreable  1 Percent  I 


Check  whether  results  apply  to: 

Local  tryout 

Field  tryout 


Phase  II  Hands-On  Component 
Summary  of  Rater  Agreement 


Instructions: 


(1)  Enter  the  percent  rater  agreement  for  each 
Performance  Measure  in  each  Scoreable  Unit. 


(2)  Circle  any  entry  less  than  80  percent. 


Scoreable  Performance  Measure 

•-'nit  II  2 1 3 1 4 1 5 1 6 I 7 I 8 19  I 10  1 11  I 12  1 13  i 14  I 15  I 16  1 17  1 18  I 19  I 20 


V 


Phase  II  Written  Component 

Tryout  of  Written  Scoreable  Unit  Against 
Performance  Test  of  the  Same  Task 

Task  title  and  number  


EXPECTED  PERFOBMAHCE  TEST 

MASTERS 

PASS  FAIL 


1 

2 

3 

4 

5 

Number 

Pass 

Proportion  expected  masters  pass 
(Number  pass/Number  of  expected 
masters)  = 


EXPECTED  PERFORMANCE  TEST 

NONMASTERS 

PASS  FAIL 


1 

2 

3 

4 

5 

Number 

Pass 

Proportion  expected  nonmasters  pass 
(Ntimber  pass/Number  of  expected 
nonmasters)  “ 


-Tir 


Instructions: 


Phase  II  Written  Component 

Summary  of  Acceptability  to  Expert  Soldiers 
(Question  Sheet  5) 


(1)  Enter  the  number  of  soldiers  on  which  the  results 
are  based  in  the  parentheses  next  to  each  Scoreable 
Unit  number. 

(2)  Enter  the  percent  soldiers  accepting  the  test  as  a 
fair  measure  of  ability. 


(3)  Circle  any  entry  less  than  80  percent 
Scoreable  I Percent  YES  | Notes 


! 

8 

9 

10 
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Itiase  II  Written  Component 


Instructions: 


Scoreable 


Summary  of  Agreement  indices  between 
Self-Ratings  and  Scoreable  Units 


(1)  Enter  the  number  of  soldiers  on  which  the  results  are 
based  In  the  parentheses  next  to  the  Scoreable  Unit 
number . 

(2)  Enter  for  each  Scoreable  Unit  the  Agreement  Index  for 
each  Item. 

(3)  Circle  any  zero  or  negative  entry. 


Item 


Phase  II  Written  Component 


Instructions: 


Scoreable 


Summary  of  Agreement  Indices  between 
Items  and  Scoreable  Units 


(1)  Enter  the  number  of  soldiers  on  which  the  results  are 
based  In  the  parentheses  next  to  the  Scoreable  Unit 
number . 

(2)  Enter  for  each  Scoreable  Unit  the  Agreement  Index  for 
each  Item. 

(3)  Circle  any  zero  or  negative  entry. 


Item 


10 


Phase  II  Written  Component 


I 


Instructions: 


Summary  Evaluation  of  Items 


(1)  Use  the  following  scale  to  Indicate  the  summary 
evaluation  for  each  Item  within  each  Scoreable 
Unit. 

A = Acceptable  (both  agreement  Indices  positive) 
B = Review  (only  one  agreement  Index  positive) 

C = Revise  (neither  agreement  index  positive) 


(2)  Enter  A,  B,  or  C for  each  item  in  each  Scoreable 


1 


ADMINISTRATION  OF  WRITTEN  SCOREABLE  UNIT  TO  ITEM  WRITER 
Task  titl_  and  number 


INITI/J.  SELF-RATING; 


TIME:  START 

FINISH 
TOTAL 


WRITER'S  RESPONSES 


DID  NOT  KNOW  SUBJECT  MATTER 


DISAGREED  WITH  KEY 


DID  NOT  UNDERSTAND  QUESTION 


DID  NOT  UNDERSTAND  ALTERNATIVES 


CONFUSED  BY  MULTI-CORRECT  ALTERNATIVES 


CONFUSED  BY  CHART  OR  ILLUSTRATION 


ALTERNATIVES  DID  NOT  REQUIRE  THOUGHT 


UESTION  CONFUSING 


ALTERNATIVES  CONFUSING 


NOT  "REAL  WORLD"  ALTERNATIVES 


IMPROPER  SUBJECT  MATTER  LEVEL 


KNOWLEDGE  NOT  CRITICAL  FOR  PERFORMANCE 


TEST  CONTAINED  INTERNAL  EVIDENCE 


ITEM  IS  SATISFACTORY 


♦IDENTIFY  RESPONSES  BY  ITEM  NUMBER  AND  RECORD  COMMENTS. 


RESPONSE/ 

ITEM 


EXPERT'S  COMMENTS 


QUESTIONS  AND/OR  COMMENTS  CONCERNING  ANY  ASPECT  OF  THE  TESTING  OR  ADMINIS- 
TRATION:   


SELF-RATING  AFTER  COMPLETION  OF  SCOREABLE  UNIT  (BEFORE  SCORING)  THE  SAME  AS 
INITIAL  SELF-RATING?  YES  NO  IF  NOT,  WHY?  


IS  THE  SCOREABLE  UNIT  CONSIDERED  TO  BE  A TRUE  MEASURE  OF  ABILITY  TO  PERFORM 
THE  TASK?  YES  NO  IF  NOT,  WHY?  
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TRYOUT  OF  WRITTEN  SCOREABLE  UNIT 


SOLDIERS 


ITEMS  TOTAL 

123456789  10  SCORE 


SUMMARY  FOim  FOR  SCOREABLE  UNITS  WITH  FOUR  OR  FEWER  ITEMS. 
Task  title  and  number 


j 

* 

1 

p 

ITEM  1 

F 

f 

P 

ITEM  2 

F 

P 

ITEM  3 

F 

ITEM  2 
P F 


ITEM  3 
P F 


ITEM  PAIRS 


Item  1 - Item  2 


Item  1 - Item  3 


Item  1 - Item  A 


Item  2 - Item  3 
Item  2 - Item  4 
Item  3 - Item  4 


AGREEMENT  INDEX 


GREATER  THAN  ZERO 


YES  NO 


